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INTRODUCTION 


Developing and releasing a user-friendly school report card site is not simply a project to be 
completed, it is a process for ongoing engagement and improvement. It takes considerable time 
and capacity; openness to meaningfully engage with stakeholders from the beginning stages 
through the report card release; and, clarity around the multiple audiences served. And it often 
requires forming key partnerships to execute design and implementation. While there are often 
deadlines for delivery, the work does not stop upon release. There are always opportunities 
following the release of a report card to continue communication and engagement with users. The 
following case study offers a glimpse into the continuous improvement efforts of three states— 
Georgia, Illinois, and Virginia—to make their school and district report cards more user-friendly. 


High-quality school and district report cards can provide a transparent picture of educational 
performance. They can be a critical tool for empowering parents, educators and the public in 
decisions about the direction of education. And they can promote a more robust conversation 
about performance beyond singular accountability ratings or assessment results. To meet these 
objectives, the report cards need to deliver data and information that users desire and in ways that 
are easy to access and understand. 


States recognize that simply putting a collection of data on a website is not enough. While 
transparency is important, state leaders have committed to deep stakeholder engagement 

to design a tool that better meets the needs of their constituents. As required by the federal 
Every Student Succeeds Act (ESSA), states must build on their previous report cards and 

publish “easily accessible and user-friendly” annual school report cards that describe the state’s 
accountability system, define its indicators, display school ratings, and present disaggregated 
student performance data’. This is not a simple task, but states are up to the challenge. As they 
demonstrated in the development of their ESSA plans, states have taken stakeholder engagement 
to new heights in this time of transition and are ready to translate that into user-friendly tools. 


The Council of Chief State School Officers (CCSSO) has offered a number of resources to support 
states in their efforts to develop and implement their school and district accountability and 
reporting systems. In May 2017, CCSSO partnered with a number of other national organizations 
to release A State Guide to Building Online Schools Report Cards. The resource suggested a 
four-phase process for developing report cards and offered additional resources to state leaders. 


Diving a level deeper, with a focus on both engagement and development of school report 
cards, in October 2017, CCSSO released Communicating Performance: A Best Practices Resource 


for Developing State Report Cards. The resource provided a recommended set of strategies 


to connect the state’s theory of action to the design of the report card; use data to tell stories; 
offer multiple engagement and feedback mechanisms; ensure high-quality data, and focus on 
continuous improvement. Finally, in March 2018, CCSSO released A School Finder to Empower: 
A Case Study of Louisiana's New Schoo! Report Card, which offered a detailed description of how 
one state went from theory to action to implementation of a new report card system. The case 


study also presented lessons learned from other states engaged in this work. 


1 For more information on ESSA report card requirement please access this resource: http://www.ccsso.org/ 
sites/default/files/2017-10/ESSAStateReportCardRequirementsMemo01262016.padf 


This case study builds upon the previous CCSSO resources to offer the experience of three more 
states in various stages of their school report card development. It highlights real-life examples 
from Georgia, Illinois, and Virginia of the recommended best practice strategies outlined in the 
previous CCSSO resources. It conveys that school and district report card production is a process, 
not a project. And this case study aims to capture additional lessons learned, beyond those shared 
in the case study of Louisiana’s new report card, that may be useful for other states as they work to 
both meet the requirements of ESSA and empower their stakeholders with actionable information 
about school performance. In particular, there are five lessons, featured below, from the report 
card development and release process of the three highlighted states: 


1. Nota One-Person Job 


Keep it Simple 


Z 
3. Educate the Agency 
4. Inputs Matter 

5 


Expect Speedbumps 


DEVELOPMENT AND IMPLEMENTATION PROCESS 


Each of the three highlighted states has focused on the following critical components in their 
development and implementation process: engaging stakeholders to identify metrics and design 
for the accountability and reporting systems; creating a compelling design that met multiple 

users’ needs; and, broadly preparing for and executing a communications strategy for release. 

Each state has developed key, ongoing partnerships in the design, production, engagement, 

and communications about their reporting systems. The states all continue to identify and work 

on improvements to their systems. In particular, the states are deeply focused on the release of 
additional accountability and contextual measures resulting from the Every Student Succeeds Act. 
Each state anticipates releasing an update to its school and district report cards by the end of 2018. 


GEORGIA 


The Georgia Department of Education (GaDOE) is currently developing an update to its school 
accountability system, known as the College and Career Readiness Performance Index (CCRPI), to 
provide a more user-friendly presentation of school, district, and state performance data. With the new 
accountability report card site, the state hopes to reframe the accountability conversation to focus on 
the key opportunities for continuous school improvement, rather than a singular focus on “chasing” 
accountability points. This means presenting a report card with greater clarity around the shared goals 
of schools, districts and the state. 


Georgia has produced its CCRPI reports for a number of years. The 2012 — 2017 reports presented a wealth 
of information on student progress, school climate, financial efficiency, and areas where schools were 
“exceeding the bar.” While the site presented a significant amount of data, stakeholders reported challenges 
in navigating and making sense of all the included information. The site simply presented data in tables 


spied Yoday jooyrs Ajpualj4-uaspy aanpolg 0} suOYA seluiBiA pue ‘sioul||| ‘2ibsoan yo Apnig asep —319a/01g JON ‘SS9I0/d 


w 


Process, Not Project - Case Study of Georgia, Illinois, and Virginia's Efforts to Produce User-Friendly School Report Cards 


aS 


and displayed calculated scores as “points” that make up the school's accountability rating. According to 
GaDOE staff, this encouraged school staff to “chase” points and stakeholders to visit other websites to get 
information about school performance in a more user-friendly and easy to understand format. 


To deliver a more usable tool, GaDOE gathered considerable user feedback and partnered with 
Tembo—a firm that specializes in data visualization, analytics, and reporting products and services—to 
create a new report card design. Based on an initial round of feedback, Tembo created a prototype of a 
new online school report card site. The Department then conducted another round of feedback, where 
they posted the prototype to their site, created an explanatory video and distributed a survey to help 
finalize the design. That feedback was turned over to the GaDOE Office of Technology Services to create 
a new public report card website and update the state’s secure data portal for administrators to keep 
consistency in design between the two systems. The newly designed report card site is slated for release 
in the fall (see Figure 1 for a draft image) 


Figure 1. Georgia’s College and Career Ready Performance Index Report. 
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Stakeholder Engagement 


During the development process of Georgia’s ESSA state plan, stakeholders’ feedback focused 
on ensuring transparent and meaningful information to the public. They provided GaDOE 

with unprompted suggestions for improving the design of its reporting system. Much of the 
accountability feedback actually related to the way the information was reported in the CCRPI 
site. As Meghan Frick, GaDOE’s Director of Communications reported, “while we [GaDOE] might 
segment out the accountability and reporting system, stakeholders don’t see it that way—it’s 

all one system to them.” Community members and educators agreed that the previous CCRPI 
reports were too tough for users to consume. Individuals reported being both overwhelmed by 
the number of indicators and the layout. Rather than helping someone understand the context of 
school performance, the site simply pointed users to a single, calculated accountability number. 


As part of its ESSA development process, the state engaged in a variety of feedback opportunities. 


GaDOE established working committees across each of the plan’s major components, including 
accountability, which brought together Department staff, educators, administrators, policymakers, 
and representatives of key stakeholder groups, such as the civil rights community. These 
committees both provided feedback to the state and reviewed other stakeholder comments. In 
particular, the Accountability Working Committee specifically considered the issue of redesigning 
the state’s accountability report cards and studied other states’ designs to offer targeted 
feedback to Tembo. To draw feedback from parents and the general public, GaDOE hosted eight 
feedback sessions around the state and released a public survey to gather additional comments. 
Additionally, the state held a Twitter chat to review specific aspects of the plan and created a 
general email account to receive comments throughout the process. 


This initial set of feedback created the foundation for the state’s redesign of the CCRPI report 
system. Stakeholders wanted the report card website to be simplified; provide more comparative 
and historical information; explain the measures in easy-to-understand terms; and, present more 
information about subgroup performance. Additionally, educators and administrators wanted the 
state to maintain consistency between the design and structure of the public report card and the 
secure data portal that they access for a deeper dive into the data. 


Design Process 


The development process occurred in three separate stages, two of which are still ongoing. The 
state decided to partner with a vendor to help develop the design of the new report card, and 
then use the Department's internal resources to build the public report card, and update its secure 
data portal to match the same design. The Georgia Department of Education engaged Tembo to 
develop a designed prototype of the new school report card website based off of the initial user 
feedback. From May to August 2017, Tembo worked with the state to create a series of images that 
could be shared with the public for further feedback. The main challenge was that the state did not 
know exactly all of the different metrics and ways to present the data that would be required given 
that their ESSA plan had not been finalized or approved. 


The state was deeply committed to ensuring that user feedback would drive the design and 
development process. Once the report card prototype was complete, GaDOE created a 2-minute 
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video to describe the new design and request feedback. State Superintendent Richard Woods 
emailed out the video and link to gather feedback to the state’s over 200,000-member listserv, which 
includes educators, administrators, parents, and the general public. Additionally, the Superintendent 
took the prototype to his Parent Advisory Council, and GaDOE held feedback groups with school 
and district representatives from across the state. Altogether, the state estimates nearly 2,000 
residents provided feedback on the initial prototype. The efforts generated quality feedback that 
indicated the state was on the right track with its new design and identified additional areas for 
improvement, such as providing more tools for analyzing multi-year performance trends. 


The GaDOE Office of Technology Services then used the prototype and comments to develop 

a test site and populate it with “dummy” data at the end of 2017. This step allowed agency staff 
to test different aspects of the system. Additionally, the Department conducted focus groups 

with select district administrators across the state to further gather feedback on the structure. 
According to GaDOE staff, the report card design evolved significantly in this period as users 
helped the state identify new tools, reorder the presentation of some of the metrics, and 
incorporate simple descriptions of the data elements. In the intervening time, Georgia’s ESSA plan 
was approved, allowing them to solidify the metrics that would be included on the report card. 


To respond to user feedback, the new report card site will use a dashboard of the main 
components of the accountability system to serve as the site’s navigation, such as student 
progress. It will present the information to allow individuals to “just get in and get the exact 
information they need” according to Nicholas Handville, Accountability Specialist at GaDOE. Each 
of the components will now include a lay person's explanation of what the measures are and why 
they are important to present. The report card will also provide users with greater information 
about the performance of demographic groups than the prior version, and enable interested users 
to download data across multiple indicators and schools for comparison. 


Communication Strategy 


While the report card site is not yet live, GaDOE has already begun to prepare internal staff and 
the school and district officials for its release. It is critical to remember that state education agency 
staff are some of the most important users of the report card. The report card can provide staff 
with meaningful information for monitoring performance across programs. And staff can be an 
important bullhorn for communicating with the field about its use. To facilitate both, according to 
Paula Swartzberg, Director of Accountability, she and her staff have “collaborated more with other 
divisions in the agency to help them understand what the message is—namely, stay away from 
chasing points—and use the data to make wise instructional decisions and inform conversations 
about opportunities and outcomes.” They are planning a demonstration during one of the 
agency's all-staff meetings, providing program-area specific training on the site, and creating a 
Frequently Asked Questions document. Additionally, they have worked with the regional education 
service agencies to help them understand the new accountability system and report card tool 
because they are ambassadors on the ground in schools, helping educators and administrators find 
and use their data for improvement.? 


2 For more information about the tools created to inform multiple stakeholder audiences, see: http://www. 
gadoe.org/Curriculum-Instruction-and-Assessment/Accountability/Pages/default.aspx. 


The Department has developed a communications plan for the report card release and has taken 
initial steps to prepare for the release. In addition to presenting at education-focused conferences 
and hosting targeted webinars through the Summer and Fall, the state is currently developing two 
videos about the report card system—one to engage and explain the goals of the report card and 
another to provide a tutorial of the features—and creating parent and principal overviews. The goal 
is to present the report card as an opportunity for all actors to learn more about a school and to 
continue the engagement about design. The state is also planning for an intensive media outreach 
to make sure that the press understands that there are two, critical stories—the data itself, and the 
new and improved way the data are presented. 


Next Steps 


One of the clearest pieces of feedback the state received from educators was that they wanted 
to maintain design consistency between the public report card and secure data portal sites. In 
previous updates to the school report card system, the state started with design changes to the 
secure portal and then pushed those to the public-facing report card site. Through this process, 
that sequencing has flipped. The state spent the previous six months working to build out a 
functional report card site that could be shared with stakeholders and tested with “dummy” data. 
As it completes that work, the Office of Technology Services is turning its attention to updating 
state’s secure data portal for educators and administrators. They are simultaneously working 

to run quality control on the newly populated data and translate final stakeholder feedback to 
the secure portal. The state can navigate the dual functions because it maintains a strong data 
governance system, which ensures that there is a single source of data that can be shared with 
multiple audiences. They anticipate releasing the public report card site and secure portal at the 
same time in fall 2018. And beyond the release of both systems, the Department will continue to 
seek out feedback from its district partners to inform improvements in the long run. 


ILLINOIS 


The Illinois State Board of Education (ISBE), in partnership with Northern Illinois University (NIU) 
through an Intergovernmental Agreement, has produced an interactive school report card 
website since 2005. In the intervening years, the website has gone through multiple updates, 
both in terms of navigation and included data. The site is intended to help families make sense 
of what is happening in schools, and is populated with numerous tools to deliver on that vision. 
Both the Data Quality Campaign and the Education Commission of the States have recognized 
Illinois for delivering a high-quality, easy-to-use product.’ The website averages just over 50,000 
unique visits per month and reached nearly 100,000 page views just after release of the new data 
in November. 


The state’s development and maintenance approach is grounded in deep partnership with 

a local university. NIU served as the external capacity to design, develop, and manage the 
website since its inception. As former Director of Illinois Interactive Report Card at NIU Harvey 
Smith reflected, the website has been a success because “we built a partnership of trust with 


3 https://2pido73em67o3eytagicp8au-wpengine.netdna-ssl.com/wp-content/uploads/2018/01/DOC-Show-Me- 
the-Data_final.odf and https://www.ecs.org/clearinghouse/01/12/53/11253.pdf 
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the agency, and it even sometimes feels like we are part of the agency.” That partnership will 
continue as the state transitions the hosting of the site to ISBE and incorporates new data related 
to the state’s ESSA plan. 


Figure 2. Illinois Interactive Report Card. 
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Stakeholder Engagement 


The goal of the report card is to provide data to parents in the most effective way for 
them. As such, the state is constantly driven to find ways to make the report card “more 
understandable, more usable, better explained and better organized,” according to Jim 
Stites, Projects Administrator for Data Strategies and Analytics at ISBE. The state has used 
both formal and informal structures to gather feedback in those areas. 


Formally, the state’s P-20 Council has a Subcommittee of Data, Assessment, and 
Accountability that is charged with identifying suggested updates for the accountability 
system and report card. The Subcommittee represents all of the major stakeholder groups in 
the state, from parents to educators to civil rights groups. ISBE and NIU also solicit feedback 
from district administrators on an ongoing basis throughout the year. Engagement with these 
groups has led to new metrics over time related to college- and career-readiness and the 
performance of English-language learners and special education students. For instance, in 
response to State Superintendent Tony Smith’s equity goals for the agency and a request 
from the field, the report now includes data on the percentage of students with Individualized 
Education Plans. The state settled on the metric through additional rounds of stakeholder 
feedback, including at a CCSSO meeting, where ISBE staff engaged with their state peers to 
share their proposed measure and demo images to gather feedback and refine. 


The state’s ESSA plan development process also served as a vehicle for identifying additional 
metrics for inclusion. Stakeholders requested more contextual information about school 
performance. As a result, for the first time, this fall the report card will include overall 
summative designations and student growth metrics. Additionally, to help highlight that 
schools sit within communities with different access to resources, the report card will include 
a “Capacity to Meet Expectations” metric that compares available school resources to the 
state-defined level of resource need. 


Finally, ISBE has drawn upon data regarding site usage to identify opportunities for 
improvement. The state tracks site traffic through Google Analytics to see which sections 
are receiving the most/least visits. Paired with information received through the site user 
feedback survey, NIU and ISBE changed the website organization to a tab structure so that 
navigation is more intuitive. The development of a set of tutorial videos and the “What's 
New” section on the homepage are also in direct response to visitor traffic and feedback. 


Design 


ISBE’s partnership with NIU for report card production dates back to 2005. NIU began 
reporting the state’s Adequate Yearly Progress designations and over time added reporting 
layers at the agency’s request. Starting with data from the 2011-12 school year, the site began 
to focus more explicitly on providing data visualizations. Throughout the years, the design has 
evolved to enable greater interaction and has benefitted from stakeholder feedback and a 
targeted review by a data visualization expert. 


The website provides many features to ease an individual’s engagement with the information 
presented. The report card landing page includes both a simple search function by school, 
district, or county name and address-based search. Seven videos walk users through the 
website to explain certain features, like the comparison tool, as well as the information 
provided, such as a detailed explanation about the school environment. The page presents 
FAQ. and links to a full data library so that users can quickly access definitions and information 
about all of the data. To help with accessibility, the entire site can be translated to Spanish. 
And to maintain their ongoing commitment to continuous improvement, the homepage 
includes a survey to gather feedback about the content and organization of the site. 
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The site provides detailed information about Academic Progress, School Environment, Students, 
Teachers, Administrators, and School Highlights. Within each heading, data are presented in a “tile” 
format that allows users to quickly scan through performance information, and select any specific 
measure for a deeper dive into the data and additional information about the specific measure. 
For instance, users can get information on students’ college readiness, postsecondary enrollment, 
and postsecondary remediation. For each measure, the state provides an (1) explanation of the 
data display, (2) context for why the measure matters, and (3) additional resources for users to learn 
more about the measure and/or recommendations for addressing the measure. One of the most 
recent additions—and the tool that has received considerable positive feedback from the field—is 
the “performance scatterplot” (see Figure 3). This visualization displays a school’s performance 

on measures such as the SAT plotted against the school’s percent low income. It provides a visual 
comparison of performance to schools in the same district and across the state. 


Figure 3. Example of Scatterplot from the Illinois Interactive Report Card. 
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High Schools in illinots 


NIU has been responsible for the development of the design of the report cards and hosting them 
on the university's servers. That is changing in the fall of 2018. Previously, ISBE produced a data 


file and transferred it to NIU for uploading and production of data visualizations. In general, that 
process has been successful to date, yet it largely resulted in a duplication of effort for both ISBE 
and NIU. Over the past year, the two staff have worked to address this issue by transitioning the 
report card production code within the ISBE data environment and are receiving data directly from 
the state’s data warehouse. This change is intended to both improve the efficiency of production 
and security of data. NIU staff are now engaged as contractors to build out the appropriate 
architecture and ensure continuity of systems. 


Communication Strategy 


As the state works to make updates to the report card to reflect its ESSA plan, ISBE is partnering 
with Advance Illinois—a nonpartisan education policy advocacy organization—on two stakeholder 
engagement sessions to gather input about what additional information the field would like to 

see and how the state can best promote the release and support the website's use. In addition, a 
manual for hosting engagement sessions was developed and distributed to advocacy groups to 
encourage additional discussion and feedback. 


The state’s standard timeline for release, as legislatively mandated, is October 31. Prior to the 
release, the state works with districts to help and encourage them to take ownership of their school 
data, both in terms of ensuring the quality of their data and understanding their performance. 

At the beginning of the month, ISBE hosts an administrator webinar to provide details about the 
release and facilitate an open forum among district leaders to discuss what they are doing to 
prepare for the release. The state identifies three or four district leaders to showcase the slides 
they intend to use with their families and local school boards, talk about how they engage families 
in the data release, and identify ways that they work with their teachers to make the data relevant 
for their continuous While districts are accustomed to this process and the general timeline, each 
year there are issues with administrators questioning their data. The sense is that too many districts 
wait until October to dig into their data, at which point it is too late to make significant changes to 
the data. State staff recognized that they could and should improve their communications through 
the data collection and quality control processes. 


To improve the process, beginning in April 2018, ISBE is publishing a monthly “Report Card 
Rundown” newsletter. It includes a “Need to Know” section that highlights upcoming changes to 
the report card and information that is relevant to administrators, such as the opening and closing 
dates for data submissions. The state has also created a new Twitter hashtag (#leadingtheway) to 
highlight districts, which offers quick stories about how districts are using the data on the report 
card to improve their practices and engage with their local communities. Additionally, the state is 
working to make data more useful in real time for teachers and administrators, which they hope will 
help improve data quality as well. 


Next Steps 


In addition to focusing on the transition of the hosting of the report card site from NIU to ISBE, 
ISBE IT team is working to empower educators and agency staff with real-time data for continuous 
improvement. Recognizing that the report card site can only provide limited use to educators 
because of the timing of the data release and the depth of data provided, the state has developed 
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a new secure data portal. The Ed360 secure data portal offers teachers and administrators with 
real-time data to make instructional decisions. The site is connected to the state’s data warehouse 
and offers the same metrics that are available on the report card, with the ability to drill down to 
student-level data. The state piloted the site with select districts starting in fall 2017 and released it 
statewide in February 2018. 


ISBE also recognized that there is a third audience that to-date has not received enough attention 
when it comes to transparent data, and that is the agency's own staff. To address this need, 

the state has rolled out a business intelligence tool that sits directly on top of the state’s data 
warehouse. The tool enables executives to be able to access basic data on their phones, while also 
allowing program staff drill down into their own data without needing to write code to query a 
database. The IT team has created dashboards and reports for specific users and is providing one- 
on-one training with each division to customize the information further. 


Staff also recognized that connected to the data reporting process there need to be increased 
professional development opportunities to ensure that those closest to students know how to use 
the tools. This will be a focus of future agency work moving forward. 


VIRGINIA 


Virginia's School Quality Profiles (GQPs) provide the public with an interactive approach to 
engaging with school and district performance. The website, released at the end of 2016, allows 
users to get a quick snapshot of performance, as well as visualize trends, filter data, and create their 
own reports. The Virginia Department of Education (VDOE) updated the state’s 15-year old school 
report cards in response to legislation that called for a more effective means of communicating 
the achievement and progress of the state’s schools. Since its release, the SOPs have received 
recognition within the state and nationally. The Richmond Technology Council awarded VDOE 

its public sector “Innovation in Utilization” award* and the state received recognition for its 

design from the National School Public Relations Association.> According to Virginia's former 

State Superintendent Steven R. Staples, “better-informed citizens drive more specific school 
improvement and | think the School Quality profiles have given us better-informed citizens.”° The 
site has been visited by more than 238,000 users since its launch in late 2016 with more than 1 
million unique page views. 


Each school site includes a data snapshot, and detailed information on accountability, assessments, 
enrollment, college & career readiness, finance, learning climate, and teacher quality. Within those 
areas, the SOP includes downloadable data tables and graphs for a variety of metrics, each of 
which includes an explanation of the metric that can be expanded for users that are interested. 

For example, the state provides data on the percent of students, by demographics that scored 
Proficient and Advanced over the past three years, compared to the district and state rates (see 
Figure 5). To help users navigate, the site header includes a link to a Frequently Asked Questions 
page and a glossary of terms that defines every included data element. 


4 http://www.doe.virginia.gov/news/news_releases/2017/05_may11.shtml 
5 http://www.doe.virginia.gov/news/news releases/2017/06 jun26.shtml| 


6 https://www.youtube.com/watch?v=XdbTvdsp-il 


Figure 4. Virginia School Quality Profile. 
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Figure 5. Student Achievement by Proficiency Level. 
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Stakeholder Engagement 


The Virginia Department of Education began publishing School Performance Report Cards online 
in the early 2000s. Those report cards were 27-page dense, static presentations of data. The 
performance data was provided absent any contextual information and was narrowly tailored to 
state- and federally-required metrics. In 2015, the Virginia General Assembly passed legislation 
directing the State Board of Education to redesign the report cards to make them more user- 
friendly and incorporate a more robust picture of school performance. House Bill 1672 required 
the State Board to conduct broad stakeholder engagement and consult with a newly established 
Standards of Learning (SOL)’ Innovation Committee, which included representatives from statewide 
stakeholder groups and legislators.® 


The State Board and VDOE partnered to host multiple roundtable sessions and conduct an online 
survey. In addition to reflecting on examples of report cards from other states, the state asked 
superintendent, principal, and teacher associations, parent-teacher association (PTA), business 
community, and civil rights organizations to provide feedback on what additional information 

the state should include going beyond simple compliance reporting. At the same time, the state 


7 Standards of Learning is Virginia’s standards and assessment system. 
8  https://lis.virginia.gov/cgi-bin/lego604.exe?151+sum+HB1672 


released a public survey to ask parents and educators what elements would be important to them 
to include. Ultimately, the degree to which individuals participate in a survey about metrics, design, 
and features can significantly impact the quality of the survey data, so Virginia invested significant 
time and capacity on the front end to ensure broad outreach. The state’s efforts paid off as it 
received over 20,000 responses on the survey, which set the stage for the initial design concept. 
Key findings from the engagement process included a desire for a tab format to easily navigate 

the site, inclusion of more information about the performance by demographic groups, and clear 
definitions for all of the data included in the site.’ 


Design Process 


While VDOE maintained strong data and information management staff capacity, the agency 
quickly realized that it would need additional support to deliver on the level of interactive features 
requested by policymakers and the public. The General Assembly allocated $250,000 for the 

initial build of the SOPs. To ensure that the Department could stay within its budget and meet 

the Legislature's timeline for release, it chose to partner with AIS Network (AISN), a Virginia- 

based technology company that was already a state-approved vendor. AISN had previous web 
development and hosting experience, yet had not previously produced a school report card. This 
meant that VDOE staff had to play a significant role in ensuring the vendor understood the specific 
requirements and framing the presentation and explanation of data. 


In particular, the staff at VDOE recognized the importance of the state being in charge of both 
its data and telling its own data story, rather than pushing those responsibilities to a vendor. 
Maintaining management of the state’s data for the reporting tool allowed the state to preserve 
flexibility, and in the long term, limit additional state expenditures. To develop that data story 
and manage the accurate delivery of data, the Directors of Communications and Information 
Management requested that the project be assigned to them collectively. As Bethann Canada 
former Director of Information Management at VDOE relayed, “this can’t be just an IT or 
communications project—it has to be both.” 


The report card development occurred in four phases. Phase 1 included the development of an 
initial “wireframe” prototype to share with the State Board and stakeholders statewide for initial 
feedback. Phase 2 included refinement of the prototype to reach Board consensus on the specific 
elements to include. Phase 3 involved the build out of the interactive features and visualizations. 
The final phase included loading the data and conducting quality control. In total, phases 1-3 took 
approximately 18 months and phase 4 took approximately 6 months. 


On the technical side, VDOE was responsible for defining requirements and handling data 
structure, suppression, and calculations. AISN receives data from the VDOE reporting database 
on a daily basis and hosts the user interface. The website was built with the goal of complete 
flexibility, so that individual metrics and data visualizations can be turned on or off with a simple 
click and VDOE staff can alter language on the site, which is built on WordPress, without having to 
alter any programming. That feature has come in particularly handy over time as VDOE works to 
simplify language and provide information in a way that is easy for a parent to understand. 


9  http://www.doe.virginia.gov/boe/meetings/2015/11_nov/agenda_items/item_e.pdf 
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Communication Strategy 


A key component of the state’s strategy for ensuring that individuals use the tool is to ensure that 
the SOPs are intuitive and simple. No matter how much training and communications support a 
state provides, if the school report card site is not user-friendly, then it is not going to be used. 

As Susan Williams, Educational Applications Manager at VDOE succinctly stated, “if it requires 
training on how to use, then it’s not the tool for us.” That does not mean that the state provides 

no support to educators and families, rather it tries to embed that into the website and in its 
general field support for exploration and use of data. The SOP site provides links to other data 
tools so that individuals that want to dig deeper into the data know where to go and allows users 
to connect each data point to social media, so that they can engage in a conversation about school 
performance with their networks, and beyond. Additionally, VDOE staff present information about 
SOP updates through webinars and at state educator and administrator conferences. Staff paid 
particular attention to the language used on the site to ensure that it was easy-to-interpret for 
parents and the public. For instance, the site originally said “school climate,” but after a discussion 
about parent-focused language at a CCSSO meeting, staff realized it could be tough to interpret as 
a parent and made the change to “learning climate.” 


The state has also worked to spread the word about its tool with the public. VDOE produced a 
YouTube video about the SOPs and has developed additional videos related to the state’s new 
accountability system under ESSA. Each year, VDOE works with local media to prepare them 

for the SOP release and answer questions about the data. And the state continues its efforts to 
gather additional feedback about ways to improve the SOPs to make them more user-friendly. As 
it develops updates for ESSA, VDOE have gone back out to the field for stakeholder engagement. 
Additionally, the state has included an option for providing feedback directly on the SOP site. 


Next Steps 


VDOE and AISN are currently working on a process to redesign the SOP site to reflect major 
changes to Virginia's state accountability system and to incorporate reporting required by the 
commonwealth’s ESSA plan. From January to April 2018, they engaged in a stakeholder review 
process to develop the new site requirements. That resulted in mock-ups for 15 web pages that 
they will hand off to the developers to implement. The goal is to release information on school 
accreditation (e.g., the state’s accountability determinations) in September and the remainder of 
the required ESSA metrics and disaggregation in December. Through this process, VDOE has 
maintained two feature and metric lists, one for what should be on the report card for the launch 
and one to indicate desired future improvements. In this way, the state recognizes that the process 
of releasing a school report card is never really done—there are always opportunities to improve. 


LESSONS LEARNED 


As other states work to develop or revise their school and district report cards, the experiences of 
Georgia, Illinois, and Virginia can be a guide. As can the previously documented lessons learned 
from Louisiana, which encouraged states to (1) put policy before presentation; (2) be clear about 


the audience; (3) partner to engage stakeholders early and often; (4) connect the report card to 
action; and, (5) focus on facilitating critical conversations.'° The following details an additional set of 
lessons learned that build upon and expand the findings from the Louisiana experience. 


1. Nota One-Person Job 


Keep it Simple 


2 
3. Educate the Agency 
4. Inputs Matter 

5 


Expect Soeedbumps 


Lesson 1: Not a One-person Job 


Designing, developing, implementing, and communicating about school and district report cards 
requires significant capacity within and beyond the state agency. It is important to remember 

that the agency is simultaneously running a large-scale technology implementation and 
communications campaign, and must staff as such. This means finding individuals that can wear 
multiple hats. As Charles Pyle, Communications Director at VDOE said, states “need to find people 
that understand both data and K-12 policy to really make this work smoothly.” 


And states need to identify appropriate partners beyond the agency to help deliver on the promise 
of a user-friendly tool. Each of the three states profiled partnered with vendors to help execute 

the design and implementation process based on their unique context. Whether the state has the 
IT capacity to turn a prototype into a full-scale reporting system as in Georgia or needs a partner 
to host the website as in Illinois and Virginia, understanding your own capacity needs and finding 
partner(s) to fill those needs is a foundational step in the process. 


Partnership goes beyond the technical execution; it is just as important when it comes to 
communication, both in terms of design and release. All three states recognized from an early 
stage that communications staff have to be at the table. Communication staff were instrumental in 
designing stakeholder engagement processes, creating and promoting public surveys, translating 
feedback to their technical counterparts, simplifying descriptive language, and developing 
materials and videos for public release. And each partnered with associations and other 
stakeholder groups to get the word out about the report card website and increase the amount of 
public feedback. 


Lesson 2: Keep it Simple 


Stakeholders across all three states delivered a consistent message—keep the report cards simple. 
Unfortunately, that simple message is not always easy to execute. Each school or district report 
card is made up of hundreds, if not thousands, of individual data points. And with the development 
of the Every Student Succeeds Act plans, states are now adding many more measures to provide a 
more complete picture of performance. 


10 https://www.ccsso.org/sites/default/files/2018-03/Louisiana%20School%20Report%20Card%20Case%20 
Study Final.odf 


spied Uoday jooyrs Ajpual4-uaspy aanpolg 0} suOyy seluiB1IA pue ‘sioull|| ‘2eibsoan yo Apnig ased —319a[01g JON ‘SSAI01d 


= 
N 


Process, Not Project - Case Study of Georgia, Illinois, and Virginia's Efforts to Produce User-Friendly School Report Cards 


= 
ee) 


Each state, in its own way, is trying to balance these competing forces. Allison Timberlake, Deputy 
Superintendent, Assessment and Accountability at GaDOE said the agency has made decisions guided 
by a single question: “how can we present this information in a way that non-education people can 
understand.” Illinois has focused its efforts on producing data visualizations because they “can present 
a clearer story than words can provide alone,” according to Jaclyn Matthews, Director of External 
Communications at the Illinois State Board of Education. And the Virginia Department of Education 

has been clear from the start that the School Quality Profiles are not the state’s single data solution. As 
Susan Williams, Educational Applications Manager at VDOE indicated, it’s the agency’s goal to “make 
all performance and context information transparent and accessible,” but they also recognize that all of 
that data does not have to fit on the state’s report card. Rather, the state is focused on identifying user 
data needs and designing appropriate tools to meet them, within and beyond the SOPs. 


Lesson 3: Educate the Agency 


Parents and educators are often cited as the most important audiences for school and district 
report cards. It is important to remember that there is a critical third audience that must be 
attended to as well—state agency staff. From the state superintendent down to the field staff, 
efforts need to be made to inform and educate internal staff about the report card data and 
functionality. Not only will many of the staff use the data in their school and district support roles, 
but they are also a critical front-line communications channel across the state. 


Accountability staff at GaDOE are working with their colleagues to be consistent in the use of their 
language, such as using the terminology achievement scores instead of proficiency rates. And the 
state is hosting a demonstration of the website at an upcoming all-staff meeting, as well as setting 
up program area specific training, to educate all employees about the system. Illinois has hired a 
full-time staff person to manage the implementation and training of employees on a new business 
intelligence tool that puts data in the hands of agency users. 


Lesson 4: Inputs Matter 


There’s a saying in information technology—garbage in, garbage out—that means no matter how 
good the internal logic of a program is, faulty inputs will produce faulty output. The same is true 
for school report cards. The accuracy of accountability ratings and other contextual information is 
dependent on the accuracy and reliability of the data. Although state agencies have long-standing 
practices for validating quality data submissions from districts, those practices are in place for 
established data collections. Under ESSA, states have both reported new measures of school 
performance and quality and are required to disaggregate data in new ways. Further, as states 
gather and report multiple years of data, inevitably new issues will arise. 


All three states confirmed their heightened concern about ensuring the quality of data in this 

time of transition. Illinois found that trying to educate district leadership on all of the new data 
changes, while also trying to have them use their data for improvement purposes was particularly 
challenging, so they created a separate process to communicate directly with leadership about all 
of the upcoming changes. Both Virginia and Georgia set aside considerable staff time and capacity 
for quality control processes in advance of report card release. And Illinois is even working to bring 
its report card in house, to limit potential errors as a result of data transactions. 


Lesson 5: Expect Soeedbumps 


No technology or stakeholder engagement process will be completely smooth—anticipate 

that and be prepared to address challenges along the way. That means both allocating 
appropriate capacity and time to execute the process. And, it means not treating development 
or engagement as a project to be completed, rather as a process that is ongoing. It is critical to 
open a consistent communication channel from the start between policy leads, data stewards, 
and programmers, and to establish clear checkpoints along the way, so that no one gets too far 
off track during development. The same can be said about stakeholder engagement. Meghan 
Frick, Director of Communications at GADOE encouraged other states to make sure they are 
“providing opportunities for feedback early and often, so that you don’t get in a position where 
you are asking for opinions, but you already know where you are going.” And she recognized 
the importance of keeping those communication channels open. Georgia can “create something 
pretty, but if people are not using it in the ways that are beneficial to them, we need to know that 
and make a change.” 


CONCLUSION / 


School and district report cards are a powerful tool for facilitating education decisions. For parents, 
they may help a decision about where to send their child. For an administrator, they may help a 
decision about where to invest additional professional development resources. For a business, 
they may help a decision about where to locate their next office. When executed in a high- 

quality manner, they help establish a common understanding of performance among educators, 
policymakers, business leaders, and the general public. And they provide a transparent platform 
for building trust in the state’s education system and efforts to improve. 


As this case study conveyed, whether a state has never produced on online school report 

card or has received awards for its report card in the past, the development and improvement 
process does not stop. Much as schools and districts are always seeking out ways to improve the 
performance of their students, so too should states seek out ways to make their data tools more 
accessible and user-friendly. And those efforts can be made better when done together. Virginia 
State Board Member Diane Atkinson attributed some of the state’s success to its willingness to 
learn from others successes and failures. As she said, “it’s always better to not reinvent the wheel.” 


The goal of delivering an interactive report card website is to empower others. As Megan Griffin, 
Director of Internal Communications at the Illinois State Board of Education, recognized, a central 
role of the state agency is to “let districts own their data” and provide appropriate supports and 

tools to realize that vision. And the same could be said about every other stakeholder in the state. 


Delivering a transparent and user-friendly system is the first step in a long line of connected actions 
to support and inspire performance. It will take partnership from actors across the state to get the 
information into the hands of those closest to students. And it will take an ongoing commitment to 
listen to their feedback and incorporate improvements over time. The process is just beginning. 
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